Method and computer readable medium for providing checkpointing to windows application groups

ABSTRACT

A computer readable medium and method providing checkpointing to Windows application groups, the computer readable medium having computer-executable instructions for execution by a processing system. The computer-executable instructions may be for launching an application and creating one or more application threads, receiving a checkpoint signal by an application thread, and entering a checkpoint Asynchronous Procedure Call (APC) handler at IRQL APC_LEVEL, the APC handler disposed in a kernel module, acquiring an ETHREAD block and user-mode context for the application thread, and examining an execution state of the one or more application threads at a time of the checkpoint signal.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present invention is a continuation of U.S. patent application Ser.No. 12/334,671 filed on Dec. 15, 2008, entitled METHOD AND SYSTEM FORPROVIDING CHECKPOINTING TO WINDOWS APPLICATION GROUPS, now issued U.S.Pat. No. 8,752,049, issued on Jun. 10, 2014, incorporated by referencein its entirety herein. The present invention is related to U.S. patentapplication Ser. No. 11/213,678 filed on Aug. 26, 2005, entitled METHODAND SYSTEM FOR PROVIDING HIGH AVAILABILITY TO COMPUTER APPLICATIONS, nowissued U.S. Pat. No. 8,122,280, issued on Feb. 21, 2012 incorporated byreference in its entirety herein.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable

NOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION

A portion of the material in this patent document is subject tocopyright protection under the copyright laws of the United States andof other countries. The owner of the copyright rights has no objectionto the facsimile reproduction by anyone of the patent document or thepatent disclosure, as it appears in the United States Patent andTrademark Office publicly available file or records, but otherwisereserves all copyright rights whatsoever. The copyright owner does nothereby waive any of its rights to have this patent document maintainedin secrecy, including without limitation its rights pursuant to 37C.F.R. §1.14.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention pertains to enterprise computer systems, computernetworks, embedded computer systems, wireless devices such as cellphones, computer systems, computers, and more particularly with methods,systems and procedures (i.e., computer readable media, software orprogramming configured to be read and/or executed by a processor on anelectronic device described or depicted herein) for providinghigh-availability, virtualization and checkpointing services for acomputer application(s) running on Microsoft Windows® Operating Systems(herein referred to as Windows or Microsoft Windows).

2. Description of the Related Art

Enterprise and wireless systems operating today are subject tocontinuous program execution that is 24 hours a day and 7 days a week.There is no longer the concept of “overnight” or “planned downtime”. Allprograms and data must be available at any point during the day andnight. Any outages or deteriorated service can result in loss of revenueas customers simply take their business elsewhere, and the enterprisestops to function on a global scale. Traditionally, achieving extremelyhigh degrees of availability has been accomplished with customizedapplications running on custom hardware, all of which is expensive andproprietary. Traditionally, no mechanisms have existed for protectingWindows applications. The problem is compounded by the fact thatMicrosoft Windows is a closed operating system without access to sourcecode, so all high availability must be provided external to theoperating system.

One reference provides a background for understanding aspects of thecurrent invention. U.S. patent application Ser. No. 11/213,678 filed onAug. 26, 2005, incorporated in its entirety, which describes how toprovide transparent and automatic high availability for applications.

BRIEF SUMMARY OF THE INVENTION

A method, system, apparatus and computer readable medium are describedfor achieving checkpointing, restoration, virtualization and loss-lessmigration of Windows applications. The invention provides transparentmigration and fail-over of Windows applications while ensuring thatconnected clients remain unaware of the migration. The client'sconnection and session are transparently transferred from the primary tothe backup server without any client involvement.

The terms “Windows” and “Microsoft Windows” is utilized hereininterchangeably to designate any and all versions of the MicrosoftWindows operating systems. By example, and not limitation, this includesWindows XP, Windows Server 2003, Windows NT, Windows Vista, WindowsServer 2008, Windows Mobile, and Windows Embedded.

The terms “checkpointer”, “checkpointing” and “checkpointing service”are utilized herein interchangeably to designate a set of serviceswhich 1) capture the entire state of an application and store all orsome of the application state locally or remotely, and 2) restore theentire state of the application from said stored application state. Thecheckpointer may include the following components: user-space dynamiclink library—the “checkpoint library”, loadable kernel module,coordinator to monitor and coordinate an application group, and a mergeutility to merge full and incremental checkpoints. The checkpointingservices run (execute) on all nodes where the application groups runs(execute) or can fail over to.

The term “checkpoint file” is utilized herein to describe the datacaptured by the checkpointing service. Generally, the checkpoint filesare written to local disk, remote disk or memory.

The term “node” is utilized herein to designate one or more processorsrunning a single instance of an operating system. A virtual machine,such as VMWare or XEN VM instance, is also considered a “node”. Using VMtechnology, it is possible to have multiple nodes on one physicalserver.

The terms “application”, “independent application”, and “windowsapplication” are utilized interchangeably herein to describe a set ofone or more processes each with one or more threads that jointly providea service. Operating systems generally launch an application by creatingthe application's initial process and letting that initial processrun/execute. In the following teachings, we often identify theapplication at launch time with that initial process and then describehow to handle creation of new processes via the CreateProcess family offunctions.

The term “application group” is utilized herein to describe a logicalgrouping of one or more independent applications that together orindependently provide some service. The independent applications do notneed to be running at the same time. A member of the application groupcan also load, perform work and exit, essentially joining and leavingthe group.

In the following, we use commonly known terms including but not limitedto “process”, “process ID (PID)”, “thread”, “thread ID (TID)”, “threadlocal storage (TLS)”, “instruction pointer”, “stack”, “kernel”, “kernelmodule”, “loadable kernel module”, “heap”, “stack”, “dish”, “CPU”, “CPUregisters”, “storage”, “memory”, “memory segments”, “memory pages”,“address space”, “semaphore”, “queues”, and “signal”. These terms arewell known in the art and thus will not be described in detail herein.

The terms “APC” and “Asynchronous Procedure Calls” are used hereininterchangeably to mean a “function that executes asynchronously in thecontext of a particular thread”. APC is provided as standardfunctionality on Microsoft operating systems, with documentationavailable by searching for “APC” on Microsoft's Developers network(msdn.microsoft.com).

The terms “CreateProcess” and “NtCreateProcess” are used to designatethe family of Microsoft Windows functions used to create new processes.The terms “TerminateProcess” and “NtTerminateProcess” are used todesignate the family of Microsoft Windows functions used to terminateprocesses. The family of functions is fully documented by Microsoft atmsdn.microsoft.com.

In the following, we also use commonly known terms and names of Windowsinternals, including but not limited to ETHREAD, EPROCESS, Alertable,NtContinue( ) IOCTL (I/O control), IRQL, and APC_LEVEL. These terms arewell known in the art and are documented by Microsoft at theirdeveloper's support network msdn.microsoft.com.

The term “coordinator” is utilized for designating a special controlprocess running as an element of the invention. The coordinator isgenerally responsible for sending out coordination events, managingapplication group registration and for coordinating activities acrossall application in an application group. For the sake of simplicity, thecoordinator is often depicted as running on the same node as theapplication group; however this is not a requirement as the coordinatorcan run on any node.

The term “transport” is utilized to designate the connection, mechanismand/or protocols used for communicating across a distributedapplication. Examples of transport include TCP/IP, Message PassingInterface (MPI), Myrinet, FiberChannel, ATM, shared memory, DMA, RDMA,system busses, and custom backplanes. In the following, the term“transport driver” is utilized to designate the implementation of thetransport. By way of example, the transport driver for TCP/IP would bethe local TCP/IP stack running on the host.

The terms “shell script” and “shelf” are used to designate the operatingsystem mechanism to run a series of commands and applications. OnWindows, shell functionality is provided by “cmd.exe” and .bat files orWindows PowerShell. Examples of cross-platform scripting technologiesinclude JavaScript, Perl, Python, and PHP. Throughout the rest of thisdocument we use the terms “shell” and “shell script” to designate thefunctionality across all scripting technologies, not just the scriptingtechnologies provided by Microsoft.

The term “interception” is used to designate the mechanism by which anapplication re-directs a system call or library call to a newimplementation. On Windows, interception can be achieved by modifying aprocess' Import Address Table and creating Trampoline functions, asdocumented by “Detours: Binary Interception of Win32 Functions” by GalenHunt and Doug Brubacher, Microsoft Research July 1999”.

The term “barrier” and “barrier synchronization” is used herein todesignate a type of synchronization method. A Barrier for a group ofprocesses and threads is a point in the execution where all threads andprocesses must stop before being allowed to proceed. Barriers aretypically implemented using semaphores, mutexes, locks, event objects,or other equivalent system functionality. Barriers are well known in theart and will not be described further here.

The term “transparent” is used herein to designate that no modificationto the application group's applications are required. In other words,the present invention works directly on the application binary withoutneeding any customization, source code modifications, recompilation,re-linking, special installation, custom agents, or other extensions.

In the following descriptions, the product name “Duration” is utilizedin referring to a system as described in the reference cited previously.It should be appreciated, however, that the teachings herein areapplicable to other similarly configured systems.

By way of example, consider an e-Commerce service consisting of aWebLogic AppServer and an Oracle Database. In this case WebLogic andOracle would be the independent applications, and the application groupwould consist of WebLogic and the Oracle database together.

By way of example, consider a cell phone with an address book andbuilt-in navigation system. In this case the address book and thenavigation system would be the independent applications, and theapplication group would consist of the address book and the navigationapplication.

By way of example, consider a shell-script running a series ofapplications and other scripts. In this case the script and allapplications and scripts launched by the script comprise the applicationgroup, and all the individual applications and other scripts calledwithin the script are the independent applications.

In at least one embodiment, the checkpointer is transparent to anapplication; no changes to the application are required.

In at least one embodiment, a method of checkpointing single processapplication groups and multi-process application groups is provided. Themethod may include creating at least one full checkpoint for eachapplication process in an application group, and may include creating atleast one incremental checkpoint for each application process in theapplication group. Further, the method may automatically merge each ofthe at least one available incremental application checkpoint against acorresponding full application checkpoint, and synchronize checkpointingacross all applications in the application group

In at least one embodiment, the checkpointer periodically or on-demandsaves an application group's data and computation state to a set ofcheckpoint files on disk. The contents of an application's checkpointfile may then be loaded at a later point in time into a new instance ofthe application, restoring the application to its previous state. Anapplication group may be restored on any machine running the sameOperating System kernel and system libraries.

In at least one embodiment, the checkpointer may include the followingapplication state in a checkpoint: process & thread attributes,execution context (instruction pointer, stack pointer, CPU registers,etc.), execution state (running, waiting, etc.), process and threadblocks, wait and mutant lists, APC queues, environment blocks, processcookie, user address space, data segments, code segment descriptors,heaps, dynamically allocated segments, file mapped segments, threadstacks, thread local storage (TLS), object, state (open files, mutexes,semaphores, events, etc.), and object handles.

In at least one embodiment, to launch applications protected by thecheckpointer, the checkpoint library is first loaded into theapplication before the application begins running. This is performedtransparently, without the need to recompile or re-link the application.The checkpoint library's initialization function is called directly whenthe library is loaded, instead of going through DllMain( ). DllMain( )is an optional entry point into a dynamic link library (DLL) that iscalled every time a thread or process is started or terminated. When athread is created, including the main thread, the thread'sinitialization routine locks a mutex before calling DllMain( ). Threadscreated within DllMain( ) would not be able to start until the mainthread exits its DllMain( ) and releases the mutex. As the checkpointlibrary needs to create its own thread, the checkpoint library isinitialized directly, as described above. Once the checkpoint libraryhas been initialized the application process' main thread jumps to theapplication's entry point, setting the application in motion.Alternatively, if an application is to be restored from a checkpoint themain thread suspends itself and the checkpoint thread initiates therestoration of the process.

In at least one embodiment, when the checkpoint library is loaded itcreates a set of objects that are used to record the data andcomputation state of the process. The checkpoint library's objects arestored in a separate heap to keep its data isolated from the rest of theapplication. Arguments are passed to the checkpointer through specificenvironment variables. For example, a restore is triggered by setting acheckpoint file to restore in the ET_CPENV_RESTORE_FILE environmentvariable. The checkpoint library then registers with the coordinator andkernel module, and creates a separate checkpoint thread. The checkpointthread is responsible for initiating the process' checkpoint, as well assaving and restoring the state of the process.

In at least one embodiment, to complete the initialization of thecheckpointer the checkpoint library installs a set of functioninterceptors. Function interceptors redirect system library calls to analternate implementation within the checkpoint library. This allows thecheckpointer to perform a number of tasks when an application calls anintercepted function, such as save and/or modify data passed between theapplication and the kernel, determine when processes, threads, andobjects are created, trap synchronization and I/O events, and preventcheckpoints during various system calls. Function interceptors are alsoused to virtualize elements of an application that cannot be restored totheir previous values, such process and thread IDs. Aside fromintercepting functions the application may call, the checkpoint libraryintercepts its process' user exception dispatcher to catch segmentationviolations triggered by the process' threads.

In at least one embodiment, the checkpointer's kernel module may performthe following tasks: Signal an application's threads to start acheckpoint, control the flow of execution during a checkpoint andrestore, and save and restore a process' kernel attributes and objects.

In at least one embodiment, the kernel module may be loaded and unloadedto/from the kernel dynamically, with no changes to the operating systemkernel being necessary. When the kernel module is loaded into the kernelit registers a named device. The checkpoint library is then able to openthis device when an application is launched. Commands are sent to thedevice via I/O control (IOCTL) codes. Upon receiving a registrationIOCTL command the kernel module creates a new process entry,creates/updates the application group's barrier, and acquires theprocess' security cookie—which is used to encrypt function pointers. Inanother embodiment, the checkpointer kernel module is built in to theoperating system kernel.

In at least one embodiment, the barrier allows the application group'sthreads to execute tasks in lock-step during a checkpoint and restore.The size of the barrier determines the number of threads that areallowed to wait at the barrier. As threads arrive at a barrier theywait, either blocking or non-blocking, for the total number of threadsto arrive. The last thread to reach the barrier then releases all thethreads waiting at the barrier. A single thread per-process may also beallowed to wait for all other threads in the process to arrive at aspecific barrier, without awakening the other threads. This not onlyallows one thread per-process to be running at any point in time, italso allows one thread to be the only thread running at a specificlocation within the checkpointer.

In at least one embodiment, the kernel module is responsible for savingand restoring the state of a process' kernel attributes. A process'kernel attributes are accessed through its EPROCESS block. The EPROCESSblock is stored in kernel space and is therefore inaccessible from userspace. An individual thread's attributes are accessed through itsETHREAD block, also inaccessible from user space. Process and threadblocks cannot be directly restored in memory because kernel-space isshared between all processes. Therefore, on restore a process andthread's kernel attributes are reconstructed by restoring each elementindividually.

In at least one embodiment, the kernel module is responsible for savingand restoring the state of a process' open kernel objects. Kernelobjects are created by user-mode processes indirectly through systemcalls. The kernel assigns each object a handle that is returned touser-space. User-mode processes then access kernel objects through theiruser-space handles. For example, if a process calls CreateFile( ) tocreate a file the kernel creates a file object in kernel space andreturns the file's handle back to the process in user space. Fileoperations are then performed by passing the file handle back to thekernel, which access the file object referenced by the handle andoperates on the object directly. Like a process' kernel attributes, onrestore a kernel object is reconstructed by restoring each attributeindividually.

In at least one embodiment, to initiate an application group'scheckpoint the coordinator wakes up each application process' checkpointthread from the checkpoint barrier. Upon awakening, each checkpointthread sends its process' application-threads a special-kernel APC viathe kernel module. Each thread is interrupted at IRQL APC_LEVEL andenters the checkpoint APC handler within the kernel module, whether ornot it was executing in user or kernel space. Before being able to savethe state of the process each of the application's threads must be in arestorable state. Because the kernel address space of a process cannotbe directly restored in memory, a thread cannot be active in kernelspace when its state is saved. An “active” thread is utilized herein tomean a thread in a running, ready, or standby state. If a thread isactive in kernel space at the time it receives the checkpoint signal aset of hooks are installed to detect when the thread arrives at arestorable state.

When all of the application's threads are in a restorable state theyproceed in lock-step throughout the checkpoint. An application's threadsare responsible for saving their own state, whereas an applicationprocess' checkpoint thread is responsible for saving the state of theprocess' attributes, objects, and user address space. The coordinator'scheckpoint thread simply saves the state of the application group'sprocess table. When an application group's checkpoint is complete eachof the applications are resumed.

It should be noted that each of the application's thread are in thekernel module when their state is saved. This does not conflict with theaforementioned algorithm; the checkpointer differentiates betweenelements of a thread's state that are a result of entering thecheckpoint APC and elements that are not.

In at least one embodiment, to restore an application group from acheckpoint the coordinator is passed a checkpoint file to restoreinstead of an application to start. The coordinator reads the processtable contained in the checkpoint and launches each of its childprocesses and all orphaned processes in the application group. Beforelaunching each process the coordinator sets the path of the process'checkpoint file in an environment variable. Upon loading, the checkpointlibrary checks for the checkpoint-file environment variable andinitializes the checkpoint library for restore if set.

On restore an application re-registers with the coordinator and kernelmodule. An application then re-launches its child processes andrecreates its threads that existed at the time of the checkpoint. Whenall threads in the application group have been recreated each processrestores their state in lock-step. Each process' checkpoint thread isresponsible for restoring the process' kernel attributes, objects, anduser address space. Unlike a process' kernel-space components, thesegments of a process' user address space are restored directly inmemory. An application's threads are responsible for restoring theirindividual state. Once the application group has been restored allapplications are resumed.

In at least one embodiment, the checkpointer only saves the elements ofa process that change after the previous checkpoint, which cansignificantly reduce the time to take a checkpoint. A process must firsttake one full checkpoint, after which all subsequent checkpoints may betaken incrementally. Each incremental checkpoint is merged with theprocess' previous full checkpoint to produce a new full checkpoint.Merging is performed by a separate merge utility, and may therefore beperformed asynchronously. The merge utility is not required to berunning on the same node as the application group.

In at least one embodiment, applications may be added to an applicationgroup in two ways. The coordinator may launch a user-specifiedapplication at any time, or an application may create a new applicationby calling CreateProcess. CreateProcess creates a new running processwith an image specified by the caller. The checkpoint library interceptsCreateProcess to launch the application with checkpointing support.Newly registered applications will be checkpointed and restored alongwith the rest of the applications in the application group.

In at least one embodiment, applications are removed from an applicationgroup upon exit. To determine when an application process exits thecheckpoint library intercepts TerminateProcess. Within theTerminateProcess interceptor the checkpointer unregisters theapplication from the coordinator before allowing the application toexit. The coordinator is then able to determine if a process has exitedin a planned or unplanned fashion depending on whether or not it hasunregistered by the time it terminates. A user-defined policy governswhether or not an application group should be brought down if one of itsprocesses exits in an unplanned fashion. Nevertheless, the coordinatorexits once every application of the group has terminated.

In at least one embodiment, checkpointing services are configured forautomatically performing a number of application services, including:injecting registration code into all applications in the applicationgroup during launch, registering the group's application as they launch,detecting execution failures, and executing from backup nodes inresponse to application group failure, application failure or nodefailure. The services can be integrated transparently into the system inthat they are implemented on the system without the need of modifying orrecompiling the application program, without the need of a customloader, or without the need for a custom operating system kernel. Inanother embodiment, a custom loader is used.

In at least one embodiment, the checkpointing services support shellscripts, where the core shell script application launches newindependent applications in any order.

The present invention comprises a set of checkpointing services forapplication groups. The checkpointing services run on every node wherethe group application can run. One embodiment of the invention generallyfunctions as an extension of the operating system and runs on all nodes.A coordination mechanism is utilized to ensure that the execution of theindependent applications is coordinated at certain points.

By way of example, and not of limitation, the present inventionimplements checkpointing services for stateless applications (e.g.,sendmail), stateful applications (e.g., Voice over IP (VOIP)),multi-tier enterprise applications (e.g., Apache, WebLogic and OracleDatabase combined), wireless devices, such as cell phones, pages andPDAs, and large distributed applications, for example those found inHigh Performance Computing (HPC), such as seismic exploration andfinancial modeling.

According to one aspect of the invention, the application group runs ona node, with one or more of the independent applications running at anypoint in time. Each independent application is running independently,but is protected and checkpointed together with all other independentapplications in the application group.

According to one aspect of the invention the application group has oneor more backup nodes ready to execute the independent application in theplace of the original in the event of a fault. The protection of theapplication group is thus coordinated and guaranteed to be consistentacross fault recovery.

An application group can be configured according to the invention withany number of independent applications. Each independent applicationruns on the primary node while the backup node for the applicationsstands ready to take over in the event of a fault and subsequentrecovery. The primary and backup can be different nodes or the primaryand backup can be the same node, in which case the fault recovery islocal.

The invention provides layered checkpointing services for applicationgroups, with checkpointing services provided both at the applicationgroup level and at the individual independent application level. Highavailability, including fault detection and recovery, for the individualindependent application is provided by Duration's existing stateful HighAvailability Services. The invention layers a distributed faultdetection and recovery mechanism on top of the local fault detection andensures that fault detection and recovery is consistent across theentire grid.

By way of example, and not of limitation, the invention implementsstateless or stateful recovery of application groups by recovering eachindependent application and ensuring all independent applications arerecovered in a consistent state. The recovery is automatic without anyapplication group or independent application involvement.

According to an aspect of the invention, there is a clean separation ofthe application logic from the checkpointing services code. This allowsapplication programmers to focus on writing their application code,rather than on writing checkpointing code. An administrator can makeapplications highly available by simply configuring the desiredsettings, such as by using a graphical configuration tool implementedaccording to the invention. The result is that high availabilityapplications are developed easily and deployed quickly without thenecessity of custom coding.

According to another aspect of the invention, protection is providedagainst node faults, network faults and process faults. The presentinvention provides user-controlled system management, automaticavailability management, and publish/subscribe event management,including notification of faults and alarms.

In various embodiments of the invention, features are provided that areuseful for application groups that must be highly available, includingbut not limited to the following:

(a) Stateful high availability and checkpointing for application groups,scripts, including high performance computing, financial modeling,enterprise applications, web servers, application servers, databases,Voice Over IP (VOIP), Session Initiation Protocol (SIP), streamingmedia, Service Oriented Architectures (SOA), wireless devices, such ascell phones, and PDA.

(b) Coordinated Restart and stateful restore for applications groups.

(c) Coordinated and transparent checkpointing of application groups.

(d) Coordinated full and incremental checkpointing for applicationsgroups.

(e) Checkpoints stored on local disks, shared disks, or memories.

(f) Automatic and transparent fault detection for application groups.

(g) Node fault detection.

(h) Process fault detection.

(i) Application group deadlock and hang protection through externalhealth checks.

(j) Coordinated automatic and transparent recovery of applicationsgroups.

(k) Auto-startup of applications groups.

(l) Script support of starting, stopping, or restarting.

(m) Dynamic policy updates.

(n) User-controllable migration of distributed applications.

The invention can be practiced according to various aspects andembodiments, including, but not limited to, those described in thefollowing aspects and embodiments which are described using phraseologywhich is generally similar to the claim language.

According to an aspect of the invention a method for achievingtransparent integration of a application group program with ahigh-availability protection program comprises: (a) injectingregistration code, transparently and automatically, into all independentapplications when they launch, without the need of modifying orrecompiling the application program and without the need of a customloader; (b) registering the independent applications automatically withthe high-availability protection program; (c) detecting a failure in theexecution of the application group or any independent application withinthe group; and (d) executing the application group with applicationgroup being executed from their respective backup servers automaticallyin response to the failure. The high-availability protection program ispreferably configured as an extension of the operating system whereinrecovery of application groups can be performed without modifyingprogramming within said application programs. The high-availabilityprotection can be configured for protecting against node faults, networkfaults, and process faults.

According to another aspect of the invention, a method, system,improvement or computer program is provided for performing loss-lessmigration of an application group, including loss-less migration of allindependent applications from their respective primary nodes to theirbackup nodes and while being transparent to a client connected to theprimary node over a TCP/IP, MPI, system bus or other transport. Thetransport, i.e. TCP/IP, MPI, or system bus will optionally be flushedand/or halted during checkpointing.

According to another aspect of the invention, a method, system,improvement or computer program performs loss-less migration of anapplication group, comprising: (a) migrating the independentapplications within an application group, without loss, from theirrespective primary nodes to at least one backup node; (b) maintainingtransparency to a client connected to the primary node over a transportconnection; (c) optionally flushing and halting the transport connectionduring the taking of checkpoints; and (d) restoring the applicationgroup, including all independent applications, from their checkpoints inresponse to initiating recovery of the application. The executiontransparency to the client is maintained by a high-availabilityprotection program configured to automatically coordinate transparentrecovery of distributed applications. Transparency is maintained by ahigh-availability protection program to said one or more independentapplications running on a primary node while at least one backup nodestands ready in the event of a fault and subsequent recovery Accordingto another aspect of the invention, a method, system, improvement orcomputer program performs fault protection for applications distributedacross multiple computer nodes, comprising: (a) providinghigh-availability application services for transparently loadingapplications, registering applications for protection, detecting faultsin applications, and initiating recovery of applications; (b) takingcheckpoints of independent applications within applications groups; (c)restoring the independent applications from the checkpoints in responseto initiating recovery of one or more the applications; (d) wherein saidhigh-availability application services are provided to the independentapplications running on a primary node, while at least one backup nodestands ready in the event of a fault and subsequent recovery; and (e)coordinating execution of individual independent applications by acoordinator program which is executed on a node accessible to themultiple computer nodes.

According to another aspect of the invention, a method, system,improvement or computer program performs loss-less migration of anapplication group, comprising: (a) a high-availability services moduleconfigured for execution in conjunction with an operating system uponwhich at least one application can be executed on one or more computernodes of a distributed system; and (b) programming within thehigh-availability services module executable on the computer nodes forloss-less migration of independent applications, (b)(i) checkpointing ofall state in the transport connection, (b)(ii) coordinatingcheckpointing of the state of the transport connection across theapplication group (b)(iii) restoring all states in the transportconnection to the state they were in at the last checkpoint, (b)(iv)coordinating recovery within a restore procedure that is coupled to thetransport connection.

According to another aspect of the invention, there is described amethod, system, improvement and/or computer program for maintaining alltransport connections across a fault. Transport connections will beautomatically restored using Duration's virtual IP addressingmechanisms.

Another aspect of the invention is a method, system, improvement and/orcomputer program that provides a mechanism to ensure that theindependent applications are launched in the proper order and with theproper timing constraints during recovery. In one embodiment, amechanism is also provided to ensure that application programs arerecovered in the proper order.

Another aspect of the invention is a method, system, computer program,computer executable program, or improvement wherein user controllablelaunch of independent applications for the application group isprovided.

Another aspect of the invention is a method, system, computer program,computer executable program, or improvement wherein user controllablestop of independent applications and application group is provided.

Further aspects of the invention will be brought out in the followingportions of the specification, wherein the detailed description is forthe purpose of fully disclosing preferred embodiments of the inventionwithout placing limitations thereon.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The invention will be more fully understood by reference to thefollowing drawings which are for illustrative purposes only:

FIG. 1 is a block diagram of how the coordinator launches the initialapplication of an application group.

FIGS. 2A and 2B are block diagrams of how an application process takes acheckpoint.

FIG. 3 is a block diagram of how an application group to restore islaunched.

FIGS. 4A and 4B are block diagrams of how an application process isrestored from a checkpoint.

FIG. 5 is a block diagram of how a new independent application joins anapplication group.

FIG. 6 is a block diagram of how a process within an application grouplaunches a new application.

FIG. 7 is a block diagram of how an application group takes full andincremental checkpoints.

FIG. 8 is a block diagram of how an application is removed from itsapplication group.

FIG. 9 is a block diagram illustrating typical deployment scenarios.

FIG. 10 is a block diagram illustrating devices and computers runningthe invention.

DETAILED DESCRIPTION OF THE INVENTION

Referring more specifically to the drawings, for illustrative purposesthe present invention will be described in relation to FIG. 1 throughFIG. 10. It will be appreciated that the system and apparatus of theinvention may vary as to configuration and as to details of theconstituent components, and that the method may vary as to the specificsteps and sequence, without departing from the basic concepts asdisclosed herein.

1. INTRODUCTION

The context in which this invention is described is a multi-processapplication group consisting of one or more threads per process. Eachapplication group runs on the primary node and can be supported by oneor more designated backup nodes. Without affecting the general case ofmultiple backups, the following describes scenarios where eachapplication group has one primary node and one backup node. Multiplebackups are handled in a similar manner as a single backup. If thebackup node is the same as the primary node, all recovery is local.

2. LAUNCHING THE INITIAL APPLICATION OF AN APPLICATION GROUP

FIG. 1 illustrates, by way of example embodiment 10, how the coordinatorlaunches the initial application of an application group. Upon starting,the coordinator 12 determines if an application group is to be startedor restored depending on its command line arguments. If a newapplication group is to be started, the coordinator creates a processtable in shared memory to store information about each process of theapplication group, registers with the kernel module to create theapplication group barrier, creates a named pipe for applications toregister and unregister, and creates a checkpoint thread 14 to initiatean application group checkpoint. The coordinator then proceeds to launchthe initial application 16 of the application group.

The coordinator first launches the application process in a suspendedstate 22. Before allowing the application to run, the coordinator fillsa buffer with machine instructions to load the checkpoint library, callthe checkpoint library's initialization function, and jump to theapplication's original entry point. The coordinator inserts the bufferinto the address space of the application and changes the application'sentry point to begin executing the first instruction contained in thebuffer 16. The application is then resumed. Checkpointer information andpolicies are passed to the application through environment variables,such as the coordinator's PID, logical name of the application group,and directory for storing checkpoints.

Upon starting, the application executes the instructions contained inthe inserted buffer to load the checkpoint library 24. The checkpointlibrary then initializes the checkpointer, which includes registeringwith the coordinator 18 and kernel module, mapping the applicationgroup's process table, creating the checkpoint thread 32, and installingfunction interceptors. This is all performed before the application hasexecuted any of its own code. Once the checkpointer has been initializedthe checkpoint library passes control to the application by instructingthe main thread to jump to the application's original entry point 26 andthe application proceeds to run.

3. CHECKPOINTING AN APPLICATION PROCESS

There are three components to taking an application checkpoint: the roleof the coordinator is described in 3a, the role of an application'scheckpoint thread is described in 3b, and the role of the applications'individual threads are described in 3c.

3a. Coordinator's Role During a Checkpoint

FIG. 7 illustrates, by way of example embodiment 280, how thecoordinator 298 triggers and coordinates an application groupcheckpoint. The coordinator's checkpoint thread 282 is responsible fortriggering an application group's checkpoint. The checkpoint threadwaits 284 until triggered by an elapsed user-defined period or by anexternal programmatic trigger, as described in the reference above. Uponawakening 286, the checkpoint thread first acquires the applicationgroup's checkpoint lock. This prevents the application group fromperforming actions that are not allowed during a checkpoint, such aslaunching new processes or creating new threads. The coordinator'scheckpoint thread then releases each application's checkpoint threadfrom the checkpoint barrier to trigger a checkpoint of each applicationin the group 288.

While each application saves their state, the checkpoint thread savesthe group's process table to the coordinator's checkpoint file 290. Theprocess table includes registration information, PID, virtual PID, andbinary pathname of each process in the application group. The name ofthe checkpoint file is a combination of the logical name of theapplication group, PID, and checkpoint count. The checkpoint thread thenwaits at the barrier for each process in the application group tocomplete their checkpoint 292. Once the application group has completedthe collective checkpoint the coordinator's checkpoint thread releasesall threads in the application group from the final barrier 294,resuming each application of the group. The checkpoint thread thenreleases the checkpoint lock and goes back to waiting until it is timeto take another checkpoint.

During a checkpoint, the coordinator's main thread waits for processregistration, un-registration, and join messages 302. However, thecheckpoint lock must be acquired before a registration orun-registration message is sent. Since the coordinator's checkpointthread acquires the checkpoint lock before starting a checkpoint,neither message will be sent during a checkpoint. Likewise, joinmessages will not be serviced without first acquiring the checkpointlock. Therefore, join requests sent during a checkpoint are servicedonce the application group's collective checkpoint has completed.

The other elements on FIG. 7 are described below.

3b. Application Checkpoint Thread's Role During a Checkpoint

FIG. 2A and FIG. 2B illustrate, by way of example embodiment 40, how anapplication's checkpoint thread takes a checkpoint. After thecoordinator 42 launches an application 44, the application process'checkpoint thread 82 enters the kernel module 84 to wait at thecheckpoint barrier 86 before starting a checkpoint. Upon being awoken 88by the coordinator, the checkpoint thread creates a new checkpoint filein a directory specified by the application group's policy. Thecheckpoint filename is a combination of the logical name of theapplication group, process' virtual PID, and checkpoint count. Thecheckpoint thread then updates the checkpoint barrier to include thetotal number of threads in the process before signaling 90 each of theapplication's threads 46 to enter the checkpoint APC/signal handler 48,which is also in kernel space. The terms “signal handler” and “APCsignal handler” are utilized herein interchangeably. The checkpointsignal is sent using a special kernel APC. The checkpoint thread thenwaits at the barrier 92 for each of the application's threads to savetheir state 68, which is saved to the checkpointer's heap within theuser address space of the process.

Once the application's threads have completed saving their state and aresuspended at the barrier 70, the checkpoint thread saves the state ofthe process' kernel attributes and objects to the checkpointer's heap,which are acquired through the process' EPROCESS block. The checkpointthread then saves the segments of the process' user address space to thecheckpoint file 94. The following list describes how the memory segmentsof a process are saved:

-   -   Thread stacks: The size and location of all thread stacks are        saved. Only the used portions are saved.    -   Code segments: Each code segment's size, location, and binary or        library pathname are saved.    -   Data segments: Each data segment's size, location, and contents        are saved.    -   Dynamically allocated segments: Each dynamically allocated        segment's size and location are saved. Only a committed region's        contents are saved.    -   Heaps: Each heap's size, location, and handle are saved. Only a        committed region's contents are saved.    -   File-mapped segments: The size, location, and handle of each        file-mapped segment are saved. Only a writable region's contents        are saved.        After writing the process' memory segments to the checkpoint        file, the checkpoint thread closes the process' checkpoint file        96 and waits at the final barrier 98 for the coordinator to        release the application group from the checkpoint. Once        released, the checkpoint thread goes back to waiting at the        checkpoint barrier 86 until it is awoken to take another        checkpoint.

3c. Application Thread's Role During a Checkpoint

FIG. 2A and FIG. 2B illustrate, by way of example embodiment 40, how thethreads of an application 44, which include the main thread and threadsthe application creates 45, take a checkpoint. When an application'sthread 46 receives the checkpoint signal it enters the checkpoint APChandler 48 in the kernel module at IRQL APC_LEVEL. Executing inkernel-mode allows a thread to directly access its kernel attributes.Once a thread enters the checkpoint APC handler it acquires its ETHREADblock and user-mode context 49. These attributes are used to determinethe thread's execution state at the time it received the checkpointsignal. Because the elements of a process must be saved in a controlledmanner, all threads sync up together by waiting at the checkpointbarrier. Once every thread has arrived at the barrier they proceed inlock-step throughout the checkpoint.

Before saving the state of the process each of the application's threadsfirst ensure they were interrupted in a restorable state. A thread mustnot save its state if it was active in kernel space at the time itreceived the checkpoint signal because its state will not be able to berestored. Unlike a process' private user address space, kernel-space isglobal and shared by all processes. Therefore, the addresses of athread's kernel components and kernel stack may not be available onrestore. Each thread determines whether or not it was interrupted in arestorable state by examining its execution state at the time of thecheckpoint signal. If a thread was either in a wait state 50 orexecuting in user-space it is already in a restorable state, andproceeds to the next checkpoint barrier. Alternatively, if a thread wasactive in kernel space 52 at the time it received the checkpoint signalit installs a set of hooks to detect when it either enters a wait stateor returns to user space.

To determine if and when a thread executing in kernel space returns touser space, the thread's user-mode instruction pointer is set to aninvalid address 54 before returning from the checkpoint signal handler56. This will generate an exception as soon as the thread returns touser space 58, which in turn will be caught by the checkpoint library'sexception dispatcher interceptor 60. The interceptor ensures acheckpoint is in progress and directs the thread to the next checkpointbarrier in the kernel module 65 as the thread has arrived in arestorable state.

To determine if and when a thread executing in kernel space enters await state, a user-mode APC is inserted at the front of the thread's APCqueue 54 before returning from the checkpoint signal handler 56. Thiswill cause the thread to enter a user-mode APC handler in the checkpointlibrary 62 if the thread proceeds to wait. The APC handler in turndirects the thread to the next checkpoint barrier in the kernel module65 as the thread has arrived in a restorable state. Note, for auser-mode APC to trigger, a thread must wait in an “alertable” state.The checkpoint library ensures this is the case by setting the Alertableflag within the interceptors of the wait functions. To prevent a threadfrom awakening prematurely from a wait state the thread did not intendto be alertable, the checkpoint library intercepts all blockingfunctions and manages each thread's user-mode APC queue.

Once all threads have arrived at the next checkpoint barrier they resettheir user-mode instruction pointer and remove the user-mode APC fromtheir APC queue 66, if still queued. Each thread then proceeds to saveits state 68 to the checkpointer's heap by accessing its ETHREAD block.A thread's state includes, but is not limited to, its user-mode context,wait and mutant lists, APC queues, and control block. When each threadfinishes saving its state it waits at the final barrier 70 for thecoordinator to release the application group from the checkpoint. Whenreleased, all the application's threads return from the kernel module 72to resume where they left off before the checkpoint. Threads caughtentering a wait state 74 resume waiting from the checkpoint library'swait interceptors 80, and threads caught exiting the kernel by theexception dispatcher interceptor 76 resume from their valid user-modecontext by calling NtContinue( ) 78.

4. RESTORING AN APPLICATION GROUP

FIG. 3 illustrates, by way of example embodiment 100, how an applicationgroup to restore is launched. On restore, the coordinator 110 islaunched with a checkpoint file to restore set in its command linearguments. The coordinator opens the coordinator's checkpoint file andrestores the application group's process table in shared memory, whichincludes the process hierarchy of the application group. The coordinatorlaunches, as described above, each of its child processes and allorphaned processes in the application group 112. In addition to thesteps described in FIG. 1 for launching an application, the coordinatorsets the path of a process' checkpoint file to restore in an environmentvariable before launching each process. Each re-launched process 122 isresponsible for launching 124 its own child processes, recreating theapplication group's process hierarchy.

5. RESTORING AN APPLICATION PROCESS FROM A CHECKPOINT

There are three components to restoring an application from acheckpoint: the role of the coordinator is described in 5a, the role ofan application's individual threads are described in 5b, and the role ofthe application's checkpoint thread is described in 5c.

5a. Coordinator's Role During Process Restore

FIG. 3 illustrates, by way of example embodiment 100, how thecoordinator 110 restores an application group. After launching thecoordinator's child processes and application group's orphaned processes112 as previously described, the coordinator's main thread closes thecheckpoint file and waits for each process of the application group tore-register 114, including processes launched by their descendants.Registration provides the coordinator with each process' new PID. Eachprocess' original, now virtual, PID is mapped to its new PID within theapplication group's process table. The coordinator's main thread thengoes back to waiting for process registration, un-registration, and joinmessages 118. Join messages will not be serviced without first acquiringthe checkpoint lock, which is owned by the checkpoint thread duringrestore. Therefore, any join requests sent during the restoration of anapplication group are not serviced until the application group has beenrestored.

Upon creation, the coordinator's checkpoint thread 102 re-acquires thecheckpoint lock and waits for each application of the group to restorefrom their checkpoint 104. Once all applications of the group have beenrestored 126 and their threads have arrived at the final barrier 128,the checkpoint thread releases all threads from the barrier 106,resuming the application group 130. The checkpoint thread then releasesthe checkpoint lock and goes back to waiting until it is time to takeanother checkpoint 108.

5b. Application Thread's Role During Restore

FIG. 4A and FIG. 4B illustrate, by way of example embodiment 140, how anapplication's threads are restored from a checkpoint. After thecoordinator 142 launches an application 144, the application's mainthread begins within the checkpoint library by checking if a restorefile is set in the environment 144. If set, the main thread re-registerswith the coordinator 142, sending it the process' new PID, registerswith the kernel module, and creates the process' checkpoint thread166—which in turn recreates the rest of the of application's threads170. The main thread then enters the kernel module 146 and waits at thebarrier for the checkpoint thread to recreate the application's threads172 that existed at the time of the checkpoint. Once each of theapplication's threads have been recreated 170, 172 and have entered thekernel module 146, all threads wait at the barrier for the checkpointthread to restore the process' kernel objects, attributes, and useraddress space 148. Once released from the barrier the application'sthreads proceed to restore their state 150.

Each thread's kernel attributes are restored individually. Addresses ofobjects referenced by a thread's ETHREAD block are changed to reflecttheir new locations in memory. Various attributes within the ETHREADblock are left alone and must not be changed, such as new thread IDs.Timed waits are also adjusted to coincide with the change in system timeand incomplete I/O operations are resumed. All threads then wait at thebarrier for the coordinator to release the application group 152. Oncereleased, each thread exits the kernel module 154 and resumes from itsuser-mode context at the time of the checkpoint. Threads previously in await state 156 resume waiting from the wait interceptor 162, and threadscaught in the exception dispatcher interceptor during a checkpoint 158resume from their valid user-mode context by calling NtContinue( ) 160.The restored application 164 proceeds to run until it is time to takeanother checkpoint.

5c. Application Checkpoint Thread's Role During Restore

FIG. 4A and FIG. 4B illustrate, by way of example embodiment 140, how anapplication's checkpoint thread restores a checkpoint. When thecheckpoint thread starts on restore it opens the checkpoint file set inthe environment. The checkpoint thread first restores the checkpointer'sheap at its previous location in the process' address space 166. Thecheckpointer's heap contains information describing the state of theprocess at the time of the checkpoint, including, but not limited to,process and thread block descriptors, kernel object descriptors, memorysegment descriptors, and list of child processes. The checkpoint threadthen re-launches the application's child processes using the same methoddescribed in FIG. 3 to launch the coordinator's child processes. Onceall child processes have been re-launched the checkpoint threadrecreates the application's threads that existed at the time of thecheckpoint 168. The application's threads are recreated by calling theCreateThread library function. Each thread's stack size and securityattributes are initialized to their previous values by settingcorresponding arguments to the function call. The thread ID table, alsolocated in the checkpointer's heap, is updated with each new thread ID.The checkpoint thread then enters the kernel module 170 and waits forthe application's threads to start and ultimately arrive at the barrier.Once all of the application's threads are suspended at the barrier thecheckpoint thread proceeds to restore the process' kernel objects,attributes, and user address space 174.

Kernel objects are recreated and their attributes are restored to theirvalues at the time of the checkpoint 174. ID attributes are updated toreflect any new IDs assigned by the kernel on restore, and objectreferences are updated to reflect their new locations in memory. Eachobject's handle is virtualized by the checkpointer so the applicationmay continue to use all its previous user-space handles. Next, theprocess' kernel attributes are restored individually. Addresses ofobjects referenced by the EPROCESS block are changed to reflect theirnew locations in memory. Various attributes within the EPROCESS blockare left alone and must not be changed, such as new process and threadIDs. The checkpoint thread then proceeds to restore the user addressspace of the process. The following list describes how the memorysegments of a process are restored:

-   -   Thread stacks: Thread stacks are restored to their original size        and location and their contents are read back into memory.    -   Code segments: Libraries loaded dynamically are reloaded.    -   Data segments: Data segments are read back into memory.    -   Dynamic segments: Dynamic memory segments are restored to their        original size and location. Each committed region is mapped back        into memory.    -   Heaps: Heaps are recreated and restored to their original size        and location. Each committed region is read back into memory.    -   File-mapped segments: Segments mapped to files are        remapped—after the process' files have been restored.        After restoring the address space of the process the checkpoint        thread releases the application's threads from the barrier and        waits for their state 176 to be restored. Once all of the        application's threads have restored their state and are        suspended at the barrier the checkpoint thread completes the        restoration of the process by restoring the process' security        cookie and reinstalling the checkpoint library's function        interceptors 177. The checkpoint thread then closes the        checkpoint file 178 and waits for the coordinator to release the        application group from the final barrier 180. Once released, the        checkpoint thread goes back to waiting at the checkpoint barrier        182 until it is awoken to take another checkpoint.

6. APPLICATION GROUP FULL AND INCREMENTAL CHECKPOINTING

FIG. 7 illustrates, by way of example embodiment 280, how an applicationgroup takes full and incremental checkpoints. As described above, thecoordinator 298 creates its checkpoint thread 282 and launches theinitial application 300. Upon launching 304, the application registerswith the coordinator 302 and creates its checkpoint thread beforeproceeding to run 306. The application's checkpoint thread 308 thenwaits at the checkpoint barrier 310 for the coordinator's checkpointthread to initiate a checkpoint. When the coordinator's checkpointthread wakes up to take a checkpoint it releases the application'scheckpoint thread 312 from the checkpoint barrier 288 and theapplication in turn proceeds to take a checkpoint.

Depending on whether or not the application process has already taken acheckpoint either a full or incremental checkpoint is taken 314. If theprocess has not yet taken a checkpoint it takes a full checkpoint 316. Afull checkpoint contains the full state of the process, as describedabove. Alternatively, if the process has already taken at least onecheckpoint it takes an incremental checkpoint 318. An incrementalcheckpoint only contains the state of the process that changed after theprevious checkpoint—except for thread stacks and checkpointer's heap,which are saved at every checkpoint. A process' kernel-elements arewritten to the checkpointer's heap during a checkpoint, as describedabove, and are therefore saved at every checkpoint.

To facilitate taking an incremental checkpoint the checkpoint librarymaintains a page table to track changes to pages belonging to a process'user address space. The checkpoint library creates a process' page tablewhen an application process is launched and populates it with theinitial pages of the process' user address space 304. All pages areinitially marked dirty. The page table is updated when pages aremodified and when pages are added and removed to/from the process'address space. The addition and removal of pages are detected byintercepting all memory allocation and de-allocation functions before anapplication is allowed to run 304, including functions that loadlibraries dynamically. When a new page is allocated the checkpointeradds a corresponding entry in the page table and marks the page dirty.Alternatively, when a page is deallocated the checkpointer removes thecorresponding entry from the page table.

To determine when an application's thread writes to a page in theprocess' user address space, the checkpointer write protects theprocess' writable user-space pages in memory after every checkpoint.Pages belonging to thread stacks and the checkpointer's heap are notwrite-protected however; they are saved at every checkpoint. When anapplication's thread writes to a write-protected page one of two actionsoccur depending on whether or not the thread is in user or kernel modeat the time. If the thread is in user mode a segmentation violationexception will be raised by the kernel. The checkpoint library catchesthe segmentation violation with its exception dispatcher interceptor andfirst verifies the page has been write-protected by the checkpointer. Ifso, the checkpointer marks the page dirty in the checkpointer's pagetable, restores the page's permissions, and then re-executes theinstruction that triggered the exception. The thread will then proceedto write to the page as it is no longer write-protected.

To handle when an application's thread writes to a write-protected pagein kernel mode, via a system call, the checkpoint library intercepts allsystem call wrappers that pass addresses of user-space buffers asarguments to a system call. If a thread writes to a write-protected pagein kernel mode the kernel will return an error. The checkpointer eitherhandles the error within the system call interceptors, or prevents sucherrors from ever occurring depending on whether or not a system call maybe re-called without changing the application's behavior. If a systemcall can be re-called without changing the application's behavior thesystem call's interceptor catches the error returned by the call,verifies the page was write-protected by the checkpointer, marks thepage dirty in the checkpointer's page table, restores the page'spermissions, and then re-executes the system call. Alternatively, if asystem call cannot be re-called without changing the application'sbehavior, the system call's interceptor restores the permissions of alluser-space pages referenced by the system call's arguments before thecall is made, marks the pages dirty in the checkpointer's page table,and then executes the system call. The system call will then proceed towrite to the user-space pages referenced by the call's arguments as theyare no longer write-protected.

When it comes time to take an incremental checkpoint the checkpointthread 308 saves the process' user-space pages that are marked dirty inthe checkpointer's page table 318. Pages belonging to thread stacks andthe checkpointer's heap are not write-protected and are therefore savedin full. Once an incremental checkpoint has been written thecheckpointer sends the incremental checkpoint and previous fullcheckpoint files to the merge utility for merging 320. Before allowingthe application to be released from the final barrier 324, thecheckpoint thread write protects the process' dirty user-space pages inmemory and clears the corresponding dirty flags in the checkpointer'spage table 322. The checkpoint thread then joins the application'sthreads at the final barrier 324 and waits for the coordinator torelease the application group from the checkpoint.

The merge utility merges an application process' incremental checkpointwith its previous full checkpoint, producing a new full checkpoint. Allpages from the incremental checkpoint are copied to the new fullcheckpoint. A page from the previous full checkpoint is copied to thenew full checkpoint if the page exists in the process' address space atthe time of the incremental checkpoint, but does not exist in theincremental checkpoint. Merging is performed asynchronously; anapplication does not need to wait for the merge to complete.

7. ADDING A NEW INDEPENDENT APPLICATION TO AN APPLICATION GROUP

FIG. 5 illustrates, by way of example embodiment 200, how a newindependent application is added to an application group. After thecoordinator 202 creates an application group by launching an initialapplication 204 as described above, the coordinator proceeds to wait forapplication process registration, un-registration, and join messages208. A join message contains a binary pathname, command line arguments,and environment variables of a new application to launch. A join messagemay be sent to the coordinator at any time.

Upon receiving a join message 207 the coordinator acquires thecheckpoint lock to prevent a checkpoint from occurring while the newapplication process is being launched. The coordinator launches theapplication 212 using the same method previously described for launchingthe initial application in FIG. 1. Once launched, the new application214 registers 216 with the coordinator. Upon receiving the registrationmessage the coordinator adds a new process entry in the applicationgroup's process table 210 and releases the checkpoint lock. Theapplication is now a member of the application group and may proceed torun 218. The coordinator then goes back to waiting for new registration,un-registration, and join requests.

8. ADDING A NEW APPLICATION LAUNCHED BY A PROCESS WITHIN AN APPLICATIONGROUP

FIG. 6 illustrates, by way of example embodiment 240, how a newapplication is launched from a process within an application group. Whenan application is launched 244 by the coordinator 242 the checkpointlibrary installs a set of function interceptors 246, includinginterceptors for CreateProcess, before the application is allowed torun. Once the application starts running 248, calls to CreateProcess areintercepted by the checkpoint library's CreateProcess interceptor 250.Within the interceptor the checkpointer first acquires the checkpointlock to prevent a checkpoint from occurring while the new applicationprocess is being launched. Before launching the new application process,the interceptor preserves data that must be saved from the currentprocess to shared memory using a named file mapping 252. Preserved dataincludes, but is not limited to, object handle tables—if handles areinherited, checkpointer policies, logical name of the application group,and the PID of the application group's coordinator. The name of the filemapping is set in an environment variable when launching the newapplication 254. The application is launched 256 using the same methodpreviously described for launching the initial application in FIG. 1. Ifthe application fails to launch an appropriate error code is set. Avalue indicating success or failure is then returned from theinterceptor 258.

On a successful launch 260 the new application's checkpoint libraryfirst checks if a named file mapping is set in the environment. If set,the checkpoint library maps the segment and copies its parent's datafrom shared memory into the process' address space 262. The checkpointlibrary then proceeds to initialize the checkpointer and register withthe coordinator 264 as previously described. Once the applicationregisters with the coordinator 242 the checkpoint lock is released. Theapplication is now a member of the application group and may proceed torun 266.

9. REMOVING AN APPLICATION FROM ITS APPLICATION GROUP

FIG. 8 illustrates, by way of example embodiment 360, how an application374 is removed from its application group. When an application islaunched 364 by the coordinator 362, the checkpoint library interceptsTerminateProcess 376 before the application is allowed to run. Once theapplication starts running 378, calls to TerminateProcess areintercepted by the checkpoint library's TerminateProcess interceptor382. Within the interceptor the checkpoint library first acquires thecheckpoint lock to prevent a checkpoint from occurring while theapplication is being removed from the application group. The checkpointlibrary then unregisters the application from the coordinator 362. Uponreceiving a un-registration message 366 the coordinator removes theprocess' entry from the application group's process table. Theapplication has now been removed from the application group. Thecheckpoint library then releases the checkpoint lock and calls the realTerminateProcess function, which then terminates the process 384. Onceall applications of the group have unregistered 368 the coordinatorunregisters from the kernel module 370 and exits 372. If the coordinatordetects an application process terminate/crash without havingunregistered the coordinator may bring down the application groupdepending on a user-defined policy.

10. LOSS-LESS MIGRATION OF APPLICATION GROUPS

Referring once again to FIG. 1 for illustrative purposes, the case ofmigrating an application group from one node to another node isconsidered. The term “migration” is utilized to mean that the runningapplication group is moved from server to server without first shuttingdown the application and the restarting the application from scratch onthe new node.

Building on the disclosures above, a loss-less migration is achieved byfirst checkpointing an application group, which includes one or moreindependent applications, and then restoring the application group on abackup node. The migration is loss-less, which means that no data orprocessing is lost.

Migration of live applications is preferably activated in theanticipation of faults, such as detecting that a CPU is overheating, orbecause a server is running out of memory. Migration may also beactivated when an administrator wants to re-configure a set of servers,or when servers currently being used have to be freed up.

11. VIRTUALIZATION AND LIVE MIGRATION OF APPLICATION GROUPS

Loss-less migration of application groups can be viewed differently. Theability to checkpoint and migrate entire application groups makes anapplication location-independent. Application groups can be moved,started and stopped on any server at any point in time. The presentteachings therefore show how to de-couple a live running instance of anapplication from the underlying operating system and hardware. Anapplication's execution has therefore been virtualized, which enableslive migration, i.e. migration of a running application, without anyapplication involvement or knowledge thereof.

12. DEPLOYMENT SCENARIOS

FIG. 9 illustrates by way of example embodiment 400 a variety of waysthe invention can be configured to operate. In one embodiment, theinvention is configured to protect a database 402, in another it isconfigured protect a pair of application servers 404, 406. In a thirdembodiment the invention is configured to protect a LAN 408 connected PC416 together with the application servers 404, 406. In a fourthembodiment the invention is configured to protect applications on a cellphone 414, which is wirelessly connected 412 to the Internet 410 theapplication servers 404,406 and the database 402. A fifth embodiment hasa home-PC 418 connected via the internet 410 to the application servers404, 406 and the LAN PC 416. The invention runs on one or more of thedevices, can be distributed across two or more of these elements, andallows for running the invention on any number of the devices(402,404,406,414,416,418) at the same time providing either a jointservice or any number of independent services.

13. SYSTEM DIAGRAM

FIG. 10 illustrates by way of example embodiment 420 a typical system422 where the invention, as described previously, can run. The systemmemory 424 can store the invention 430 as well as any runningapplication 426, 428 being protected. The system libraries 432 andoperating system 434 provide the necessary support. Local or remotestorage 436 provides persistent storage of and for the invention. Theinvention is generally loaded from storage 436 into memory 424 as partof normal operation. One or more CPUs 442 performs these functions, andmay uses the network devices 438, to access the network 444, andInput/Output devices 440.

14. CONCLUSION

In the embodiments described herein, an example programming environmentwas described for which an embodiment of programming according to theinvention was taught. It should be appreciated that the presentinvention can be implemented by one of ordinary skill in the art usingdifferent program organizations and structures, different datastructures, and of course any desired naming conventions withoutdeparting from the teachings herein. In addition, the invention can beported, or otherwise configured for, use across a wide-range ofoperating system environments.

Although the description above contains many details, these should notbe construed as limiting the scope of the invention but as merelyproviding illustrations of some of the exemplary embodiments of thisinvention. Therefore, it will be appreciated that the scope of thepresent invention fully encompasses other embodiments which may becomeobvious to those skilled in the art, and that the scope of the presentinvention is accordingly to be limited by nothing other than theappended claims, in which reference to an element in the singular is notintended to mean “one and only one” unless explicitly so stated, butrather “one or more.” All structural and functional equivalents to theelements of the above-described preferred embodiment that are known tothose of ordinary skill in the art are expressly incorporated herein byreference and are intended to be encompassed by the present claims.Moreover, it is not necessary for a device or method to address each andevery problem sought to be solved by the present invention, for it to beencompassed by the present claims. Furthermore, no element, component,or method step in the present disclosure is intended to be dedicated tothe public regardless of whether the element, component, or method stepis explicitly recited in the claims. No claim element herein is to beconstrued under the provisions of 35 U.S.C. 112, sixth paragraph, unlessthe element is expressly recited using the phrase “means for.”

What is claimed is:
 1. A non-transitory computer readable medium havingcomputer-executable instructions which, when executed by a processor,cause a processing system to perform: registering a coordinator processwith a checkpoint kernel module; launching a plurality of independentapplications, each including a plurality of threads, as an applicationgroup via the coordinator, wherein launching an application via thecoordinator causes the launched application to load a user-spacecheckpoint library; wherein an application loading the checkpointlibrary comprises installing a plurality of function interceptorsincluding at least wait function interceptors which together areconfigured to intercept system calls made by the threads of the loadingapplication during execution, and, in response to intercepting a systemcall from a calling thread and determining the coordinator has initiateda group checkpoint, are configured to cause the calling thread to blockand wait in an alertable state; initiating a group checkpoint of theapplication group via the coordinator causing each of the independentapplications to send a kernel-mode checkpoint signal to each thread ofthe application; and responsive to receiving the kernel-mode checkpointsignal, the receiving thread performs steps comprising: determining thereceiving thread was active in kernel-space at the time the kernel-modesignal was received utilizing kernel attributes directly accessible froma checkpoint signal handler; and inserting a user-mode checkpoint signalat the front of a signal queue of the receiving thread and returningfrom the checkpoint signal handler in response to the determining thatthe receiving thread was active in kernel-space, wherein a user-modesignal is only processed by a signaled thread when the signaled threadwaits in the alertable state.
 2. The non-transitory computer readablemedium of claim 1, wherein performing the application checkpoint furthercomprises directing the receiving thread to wait at a first checkpointbarrier in the kernel module in response to determining that thereceiving thread was not active in kernel-space at the time thekernel-mode signal was received.
 3. The non-transitory computer readablemedium of claim 2, wherein the user-mode checkpoint signal is configuredto cause the thread to enter a usermode signal handler in the checkpointlibrary when processed, wherein the user-mode signal handler isconfigured to direct the thread processing the usermode signal to waitat the first checkpoint barrier.
 4. The non-transitory computer readablemedium of claim 3, wherein performing the application checkpoint furthercomprises, after all threads of the application are waiting at the firstcheckpoint barrier, releasing all of threads from the first checkpointbarrier which each proceed to save their respective thread state.
 5. Thenon-transitory computer readable medium of claim 4, wherein a threadsaving its thread state comprises saving the thread state to a heap ofan application checkpoint thread.
 6. The non-transitory computerreadable medium of claim 4, wherein the threads of the applicationperforming the application checkpoint wait at a final checkpoint barrierafter saving their respective thread states.
 7. The non-transitorycomputer readable medium of claim 6, wherein the coordinator processreleases the threads waiting at the final checkpoint barrier in responseto determining that all threads of each of the independent applicationsare waiting at the final checkpoint barrier.
 8. The non-transitorycomputer readable medium of claim 1, wherein the kernel-mode checkpointsignal comprises a kernel Asynchronous Procedure Call (APC) and theuser-mode checkpoint signal comprises a user APC.
 9. The non-transitorycomputer readable medium of claim 1, wherein performing the applicationcheckpoint further comprises setting a user-mode instruction pointer ofthe receiving thread to an invalid address prior to returning from thecheckpoint signal handler in response to the determining that thereceiving thread was active in kernel-space at the time the kernel-modecheckpoint signal was received.
 10. The non-transitory computer readablemedium of claim 9, wherein the user-mode instruction pointer of thereceiving thread being set to the invalid address will generate anexception in response to the receiving thread returning to user-spaceafter returning from the checkpoint handler, wherein the exception iscaught by an exception interceptor of the checkpoint library configuredto direct the receiving thread to wait at a first checkpoint barrier.11. A method, comprising: registering a coordinator process with acheckpoint kernel module; launching a plurality of independentapplications, each including a plurality of threads, as an applicationgroup via the coordinator, wherein launching an application via thecoordinator causes the launched application to load a user-spacecheckpoint library; wherein an application loading the checkpointlibrary comprises installing a plurality of function interceptorsincluding at least wait function interceptors which together areconfigured to intercept system calls made by the threads of the loadingapplication during execution, and, in response to intercepting a systemcall from a calling thread and determining the coordinator has initiateda group checkpoint, are configured to cause the calling thread to blockand wait in an alertable state; initiating a group checkpoint of theapplication group via the coordinator causing each of the independentapplications to send a kernel-mode checkpoint signal to each thread ofthe application; and responsive to receiving the kernel-mode checkpointsignal, the receiving thread performs steps comprising: determining thereceiving thread was active in kernel-space at the time the kernel-modesignal was received utilizing kernel attributes directly accessible froma checkpoint signal handler; and inserting a user-mode checkpoint signalat the front of a signal queue of the receiving thread and returningfrom the checkpoint signal handler in response to the determining thatthe receiving thread was active in kernel-space, wherein a user-modesignal is only processed by a signaled thread when the signaled threadwaits in the alertable state.
 12. The method of claim 11, whereinperforming the application checkpoint further comprises directing thereceiving thread to wait at a first checkpoint barrier in the kernelmodule in response to determining that the receiving thread was notactive in kernel-space at the time the kernel-mode signal was received.13. The method of claim 12, wherein the user-mode checkpoint signal isconfigured to cause the thread to enter a user-mode signal handler inthe checkpoint library when processed, wherein the user-mode signalhandler is configured to direct the thread processing the user-modesignal to wait at the first checkpoint barrier.
 14. The method of claim13, wherein performing the application checkpoint further comprises,after all threads of the application are waiting at the first checkpointbarrier, releasing all of threads from the first checkpoint barrierwhich each proceed to save their respective thread state.
 15. The methodof claim 14, wherein a thread saving its thread state comprises savingthe thread state to a heap of an application checkpoint thread.
 16. Themethod of claim 14, wherein the threads of the application performingthe application checkpoint wait at a final checkpoint barrier aftersaving their respective thread states.
 17. The method of claim 16,wherein the coordinator process releases the threads waiting at thefinal checkpoint barrier in response to determining that all threads ofeach of the independent applications are waiting at the final checkpointbarrier.
 18. The method of claim 11, wherein the kernel-mode checkpointsignal comprises a kernel Asynchronous Procedure Call (APC) and theuser-mode checkpoint signal comprises a user APC.
 19. The method ofclaim 11, wherein performing the application checkpoint furthercomprises setting a user-mode instruction pointer of the receivingthread to an invalid address prior to returning from the checkpointsignal handler in response to the determining that the receiving threadwas active in kernel-space at the time the kernel-mode checkpoint signalwas received.
 20. The method of claim 19, wherein the user-modeinstruction pointer of the receiving thread being set to the invalidaddress will generate an exception in response to the receiving threadreturning to user-space after returning from the checkpoint handler,wherein the exception is caught by an exception interceptor of thecheckpoint library configured to direct the receiving thread to wait ata first checkpoint barrier.